{ "cells": [ { "cell_type": "markdown", "id": "ea63cd0b-9873-4afd-815c-35a056bc0e37", "metadata": {}, "source": [ "# Order Latency Data\n", "\n", "To obtain more realistic backtesting results, accounting for latencies is crucial. Therefore, it's important to collect both feed data and order data with timestamps to measure your order latency. The best approach is to gather your own order latencies. You can collect order latency based on your live trading or by regularly submitting orders at a price that cannot be filled and then canceling them for recording purposes. However, if you don't have access to them or want to establish a target, you will need to artificially generate order latency. You can model this latency based on factors such as feed latency, trade volume, and the number of events. In this guide, we will demonstrate a simple method to generate order latency from feed latency using a multiplier and offset for adjustment." ] }, { "cell_type": "markdown", "id": "ce73ae54-2e57-489e-919e-d1f9271f4461", "metadata": {}, "source": [ "First, loads the feed data." ] }, { "cell_type": "code", "execution_count": 1, "id": "ab04f05e-259b-4a68-b701-1df3f2c19a82", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([(3758096386, 1580515202342000000, 1580515202497052000, 9364.51, 1.197, 0, 0, 0.),\n", " (3758096386, 1580515202342000000, 1580515202497346000, 9365.67, 0.02 , 0, 0, 0.),\n", " (3758096386, 1580515202342000000, 1580515202497352000, 9365.86, 0.01 , 0, 0, 0.),\n", " ...,\n", " (3489660929, 1580601599836000000, 1580601599962961000, 9351.47, 3.914, 0, 0, 0.),\n", " (3489660929, 1580601599836000000, 1580601599963461000, 9397.78, 0.1 , 0, 0, 0.),\n", " (3489660929, 1580601599848000000, 1580601599973647000, 9348.14, 3.98 , 0, 0, 0.)],\n", " dtype=[('ev', '\n", "shape: (27_532_602, 8)
evexch_tslocal_tspxqtyorder_idivalfval
i64i64i64f64f64u64i64f64
3758096386158051520234200000015805152024970520009364.511.197000.0
3758096386158051520234200000015805152024973460009365.670.02000.0
3758096386158051520234200000015805152024973520009365.860.01000.0
3758096386158051520234200000015805152024973570009366.360.002000.0
3758096386158051520234200000015805152024973630009366.360.003000.0
……………………
3489660929158060159981200000015806015999444040009397.790.0000.0
3489660929158060159982600000015806015999521760009354.84.07000.0
3489660929158060159983600000015806015999629610009351.473.914000.0
3489660929158060159983600000015806015999634610009397.780.1000.0
3489660929158060159984800000015806015999736470009348.143.98000.0
" ], "text/plain": [ "shape: (27_532_602, 8)\n", "┌────────────┬─────────────────────┬────────────────────┬─────────┬───────┬──────────┬──────┬──────┐\n", "│ ev ┆ exch_ts ┆ local_ts ┆ px ┆ qty ┆ order_id ┆ ival ┆ fval │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ i64 ┆ i64 ┆ i64 ┆ f64 ┆ f64 ┆ u64 ┆ i64 ┆ f64 │\n", "╞════════════╪═════════════════════╪════════════════════╪═════════╪═══════╪══════════╪══════╪══════╡\n", "│ 3758096386 ┆ 1580515202342000000 ┆ 158051520249705200 ┆ 9364.51 ┆ 1.197 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3758096386 ┆ 1580515202342000000 ┆ 158051520249734600 ┆ 9365.67 ┆ 0.02 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3758096386 ┆ 1580515202342000000 ┆ 158051520249735200 ┆ 9365.86 ┆ 0.01 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3758096386 ┆ 1580515202342000000 ┆ 158051520249735700 ┆ 9366.36 ┆ 0.002 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3758096386 ┆ 1580515202342000000 ┆ 158051520249736300 ┆ 9366.36 ┆ 0.003 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 3489660929 ┆ 1580601599812000000 ┆ 158060159994440400 ┆ 9397.79 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3489660929 ┆ 1580601599826000000 ┆ 158060159995217600 ┆ 9354.8 ┆ 4.07 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3489660929 ┆ 1580601599836000000 ┆ 158060159996296100 ┆ 9351.47 ┆ 3.914 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3489660929 ┆ 1580601599836000000 ┆ 158060159996346100 ┆ 9397.78 ┆ 0.1 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "│ 3489660929 ┆ 1580601599848000000 ┆ 158060159997364700 ┆ 9348.14 ┆ 3.98 ┆ 0 ┆ 0 ┆ 0.0 │\n", "│ ┆ ┆ 0 ┆ ┆ ┆ ┆ ┆ │\n", "└────────────┴─────────────────────┴────────────────────┴─────────┴───────┴──────────┴──────┴──────┘" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import polars as pl\n", "\n", "df = pl.DataFrame(data)\n", "df" ] }, { "cell_type": "markdown", "id": "0a2f5db2-b6ab-420d-b5f9-0ca8e123607b", "metadata": {}, "source": [ "Selects only the events that have both a valid exchange timestamp and a valid local timestamp to get feed latency." ] }, { "cell_type": "code", "execution_count": 3, "id": "349335b0-50a4-4e3a-a721-c55388d71b01", "metadata": {}, "outputs": [], "source": [ "from hftbacktest import EXCH_EVENT, LOCAL_EVENT\n", "\n", "df = df.filter((pl.col('ev') & EXCH_EVENT == EXCH_EVENT) & (pl.col('ev') & LOCAL_EVENT == LOCAL_EVENT))" ] }, { "cell_type": "markdown", "id": "e641eb14-5370-4cc3-bf52-e940d94745a1", "metadata": {}, "source": [ "Reduces the number of rows by resampling to approximately 1-second intervals." ] }, { "cell_type": "code", "execution_count": 4, "id": "577ad180-42ad-46f8-9723-1b2c9b14d9c2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (86_394, 2)
exch_tslocal_ts
i64i64
15805152028430000001580515202979365000
15805152035510000001580515203943566000
15805152037890000001580515204875639000
15805152041270000001580515205962135000
15805152047380000001580515206983780000
……
15806015958690000001580601595997115000
15806015968650000001580601596994060000
15806015978640000001580601597987786000
15806015988700000001580601598997068000
15806015998480000001580601599973647000
" ], "text/plain": [ "shape: (86_394, 2)\n", "┌─────────────────────┬─────────────────────┐\n", "│ exch_ts ┆ local_ts │\n", "│ --- ┆ --- │\n", "│ i64 ┆ i64 │\n", "╞═════════════════════╪═════════════════════╡\n", "│ 1580515202843000000 ┆ 1580515202979365000 │\n", "│ 1580515203551000000 ┆ 1580515203943566000 │\n", "│ 1580515203789000000 ┆ 1580515204875639000 │\n", "│ 1580515204127000000 ┆ 1580515205962135000 │\n", "│ 1580515204738000000 ┆ 1580515206983780000 │\n", "│ … ┆ … │\n", "│ 1580601595869000000 ┆ 1580601595997115000 │\n", "│ 1580601596865000000 ┆ 1580601596994060000 │\n", "│ 1580601597864000000 ┆ 1580601597987786000 │\n", "│ 1580601598870000000 ┆ 1580601598997068000 │\n", "│ 1580601599848000000 ┆ 1580601599973647000 │\n", "└─────────────────────┴─────────────────────┘" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = df.with_columns(\n", " pl.col('local_ts').alias('ts')\n", ").group_by_dynamic(\n", " 'ts', every='1000000000i'\n", ").agg(\n", " pl.col('exch_ts').last(),\n", " pl.col('local_ts').last()\n", ").drop('ts')\n", "\n", "df" ] }, { "cell_type": "markdown", "id": "7834be83-788e-48ae-92e9-0f01bda90cd5", "metadata": {}, "source": [ "Converts back to the structured NumPy array." ] }, { "cell_type": "code", "execution_count": 5, "id": "d69e7709-d947-4e62-b5a4-c8f1e623de2a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([(1580515202843000000, 1580515202979365000),\n", " (1580515203551000000, 1580515203943566000),\n", " (1580515203789000000, 1580515204875639000), ...,\n", " (1580601597864000000, 1580601597987786000),\n", " (1580601598870000000, 1580601598997068000),\n", " (1580601599848000000, 1580601599973647000)],\n", " dtype=[('exch_ts', '\n", "shape: (86_394, 4)
req_tsexch_tsresp_ts_padding
i64i64i64i64
1580515202979365000158051520352482500015805152039339200000
1580515203943566000158051520551383000015805152066915280000
1580515204875639000158051520922219500015805152124821120000
1580515205962135000158051521330267500015805152188080800000
1580515206983780000158051521596690000015805152227042400000
…………
1580601595997115000158060159650957500015806015968939200000
1580601596994060000158060159751030000015806015978974800000
1580601597987786000158060159848293000015806015988542880000
1580601598997068000158060159950534000015806015998865440000
1580601599973647000158060160047623500015806016008531760000
" ], "text/plain": [ "shape: (86_394, 4)\n", "┌─────────────────────┬─────────────────────┬─────────────────────┬──────────┐\n", "│ req_ts ┆ exch_ts ┆ resp_ts ┆ _padding │\n", "│ --- ┆ --- ┆ --- ┆ --- │\n", "│ i64 ┆ i64 ┆ i64 ┆ i64 │\n", "╞═════════════════════╪═════════════════════╪═════════════════════╪══════════╡\n", "│ 1580515202979365000 ┆ 1580515203524825000 ┆ 1580515203933920000 ┆ 0 │\n", "│ 1580515203943566000 ┆ 1580515205513830000 ┆ 1580515206691528000 ┆ 0 │\n", "│ 1580515204875639000 ┆ 1580515209222195000 ┆ 1580515212482112000 ┆ 0 │\n", "│ 1580515205962135000 ┆ 1580515213302675000 ┆ 1580515218808080000 ┆ 0 │\n", "│ 1580515206983780000 ┆ 1580515215966900000 ┆ 1580515222704240000 ┆ 0 │\n", "│ … ┆ … ┆ … ┆ … │\n", "│ 1580601595997115000 ┆ 1580601596509575000 ┆ 1580601596893920000 ┆ 0 │\n", "│ 1580601596994060000 ┆ 1580601597510300000 ┆ 1580601597897480000 ┆ 0 │\n", "│ 1580601597987786000 ┆ 1580601598482930000 ┆ 1580601598854288000 ┆ 0 │\n", "│ 1580601598997068000 ┆ 1580601599505340000 ┆ 1580601599886544000 ┆ 0 │\n", "│ 1580601599973647000 ┆ 1580601600476235000 ┆ 1580601600853176000 ┆ 0 │\n", "└─────────────────────┴─────────────────────┴─────────────────────┴──────────┘" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_order_latency = pl.DataFrame(order_latency)\n", "df_order_latency" ] }, { "cell_type": "markdown", "id": "0c0eb2a2-64dd-4d4a-93f6-0c52a4bdc142", "metadata": {}, "source": [ "Checks if latency has invalid negative values." ] }, { "cell_type": "code", "execution_count": 8, "id": "542eace5-590e-4d30-a552-0cf0825885d5", "metadata": {}, "outputs": [], "source": [ "order_entry_latency = df_order_latency['exch_ts'] - df_order_latency['req_ts']\n", "order_resp_latency = df_order_latency['resp_ts'] - df_order_latency['exch_ts']" ] }, { "cell_type": "code", "execution_count": 9, "id": "a64aad08-888f-48e7-85cc-1719ba3333b6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(order_entry_latency <= 0).sum()" ] }, { "cell_type": "code", "execution_count": 10, "id": "7e770e94-4660-4b0b-827b-951a046d9529", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(order_resp_latency <= 0).sum()" ] }, { "cell_type": "markdown", "id": "18944b3a-747c-4c38-96e1-0ab6930e098c", "metadata": {}, "source": [ "Here, we wrap the entire process into a method with `njit` for increased speed." ] }, { "cell_type": "code", "execution_count": 11, "id": "15b080d5-9e5f-4124-a83f-813bb2ff9414", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from numba import njit\n", "import polars as pl\n", "from hftbacktest import LOCAL_EVENT, EXCH_EVENT\n", "\n", "@njit\n", "def generate_order_latency_nb(data, order_latency, mul_entry, offset_entry, mul_resp, offset_resp):\n", " for i in range(len(data)):\n", " exch_ts = data[i].exch_ts\n", " local_ts = data[i].local_ts\n", " feed_latency = local_ts - exch_ts\n", " order_entry_latency = mul_entry * feed_latency + offset_entry\n", " order_resp_latency = mul_resp * feed_latency + offset_resp\n", "\n", " req_ts = local_ts\n", " order_exch_ts = req_ts + order_entry_latency\n", " resp_ts = order_exch_ts + order_resp_latency\n", "\n", " order_latency[i].req_ts = req_ts\n", " order_latency[i].exch_ts = order_exch_ts\n", " order_latency[i].resp_ts = resp_ts\n", "\n", "def generate_order_latency(feed_file, output_file = None, mul_entry = 1, offset_entry = 0, mul_resp = 1, offset_resp = 0):\n", " data = np.load(feed_file)['data']\n", " df = pl.DataFrame(data)\n", " \n", " df = df.filter(\n", " (pl.col('ev') & EXCH_EVENT == EXCH_EVENT) & (pl.col('ev') & LOCAL_EVENT == LOCAL_EVENT)\n", " ).with_columns(\n", " pl.col('local_ts').alias('ts')\n", " ).group_by_dynamic(\n", " 'ts', every='1000000000i'\n", " ).agg(\n", " pl.col('exch_ts').last(),\n", " pl.col('local_ts').last()\n", " ).drop('ts')\n", " \n", " data = df.to_numpy(structured=True)\n", "\n", " order_latency = np.zeros(len(data), dtype=[('req_ts', 'i8'), ('exch_ts', 'i8'), ('resp_ts', 'i8'), ('_padding', 'i8')])\n", " generate_order_latency_nb(data, order_latency, mul_entry, offset_entry, mul_resp, offset_resp)\n", "\n", " if output_file is not None:\n", " np.savez_compressed(output_file, data=order_latency)\n", "\n", " return order_latency" ] }, { "cell_type": "code", "execution_count": 12, "id": "09772349-cebb-4698-8c6f-b77cbd19f15b", "metadata": {}, "outputs": [], "source": [ "order_latency = generate_order_latency('btcusdt_20200201.npz', output_file='feed_latency_20200201.npz', mul_entry=4, mul_resp=3)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 5 }